Parameter-Efficient Deep Neural Networks With Bilinear Projections

Recent research on deep neural networks (DNNs) has primarily focused improving the model accuracy. Given a proper learning framework, it is generally possible to increase depth or layer width achieve higher level of However, huge number parameters imposes more computational and memory usage overhead leads parameter redundancy. In this paper, we address redundancy problem in DNNs by replacing conventional full projections with bilinear projections. For fully-connected $D$ input nodes output nodes, applying projection can reduce space complexity from $\mathcal{O}(D^2)$ $\mathcal{O}(2D)$, achieving sub-linear size. structured lower freedom degree compared projection, causing under-fitting problem. So simply scale up mapping size increasing channels, which keep even boosts This makes very parameter-efficient handy deploy such models mobile systems limitations. Experiments four benchmark datasets show that proposed accuracies than DNNs, while significantly reduces